Skip to content

snapshot: p2p fetcher, agreement and download (P4) - #104

Merged
EnzinoBB merged 10 commits into
masterfrom
feat/snapshot-p2p-fetcher
Oct 25, 2026
Merged

EnzinoBB merged 10 commits into
masterfrom
feat/snapshot-p2p-fetcher

Conversation

@EnzinoBB

@EnzinoBB EnzinoBB commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Step P4 of docs/plans/2026-10-25-snapshot-p2p-plan.md. Based on #102 (P3): until #102 is merged the diff includes its commits.

What it does

  • snapshot_agreement.hpp/.cpp: netPrefix(address) (/24 for IPv4, /48 for IPv6, IPv4-mapped counted as IPv4, port ignored) and agree(sources, minSources). Sources are grouped by (seq, head_hash, state_digest, settings_digest) and counted by address prefix; trusted keys (sources=) count once each by key; a source whose address is unknown does not count. Among the groups that reach min_sources, the choice goes in this order: a group holding a configured key, then the most independent sources, then the highest block, which only breaks ties. A later block alone never wins over more support, so min_sources forged sources at a made-up later block lose to a larger honest group. Two groups equal on all three are a conflict, and nothing is chosen. Every group that is not chosen carries the reason, which is logged. This differs from the design's 6.3 rule "highest seq first" (a security review finding).
  • snapshot_fetcher.hpp/.cpp: cs::SnapshotFetcher, single-threaded, everything external through Hooks (send manifest/piece request, add neighbours, address lookup, ms clock, optional log sink).
    • Discover/Collect: peers with capability bit 24, sources= keys and sources_file keys are passed to addNeighbours (once per key, again after 60 s if the key is still not a neighbour). Configured keys have their own room in the source table. Discovered peers are limited to 4 per /24 (/48), 16 per /16 (/32) and 64 in all. A full table always makes room, removing the least useful entry first: not a neighbour, then silent (two requests unanswered over more than 35 s, counted from the first request; silence is a reason only when room is needed), then no manifest after a minute, then the most represented prefix. It never removes a configured source, one with pieces outstanding, or one in the agreed group during a fetch. An evicted key is not taken again for 1 min. A request without an answer is repeated after 10 s twice, then every 30 s (design 5.3a). An address that is not known yet is not capped at discovery: it is resolved again when the manifest arrives, and both the limits and the agreement count use that address; every serving neighbour is asked for its manifest every 30 s (no answer means ask again 30 s later); a manifest is kept only if fromJson, verify(), node_key == sender and the chunk size (power of two, at most 1 MiB) all pass;
    • Agreement runs at each tick, so replies to one round of requests count together. A line is logged every 30 s; when fetch_wait runs out, sync goes to Fallback with the design's warning line, and wait keeps waiting with a line every 5 min.
    • Fetch: the files of one signed manifest from the agreed group (the file list shared by most sources). Each file is fetched from every agreed source that declares the same bytes, round-robin over sources, at most 2 pieces per source and 8 in total. Other rules: a piece timeout of 30 s plus the bytes outstanding to the source at 64 KB/s (a source that timed out but delivered in the last 60 s is not marked: it gets one piece at a time until it delivers two in time, and its late reply is still taken while the piece is missing); busy waits retry_ms (at least 500 ms, at most 10 s), and busy for a minute without a piece leaves the source alone for a minute; not serving, unknown or timeout marks the source for 60 s, and a second mark keeps it out for the rest of the run; a bad chunk keeps the source out for the rest of the run and the piece is requested elsewhere; a reply that does not match an outstanding request is dropped and counted. Pieces are written at their offset into pre-sized files. When a file has no usable source left, the fetcher switches to another file group, and goes back to Collect if none is left. A fetch always ends: with nothing to ask for 2 min, or no new byte for 5 min, it goes back to Collect, and counts as a failed fetch unless a source was banned in it for bytes that do not match the signed manifest (timeouts, busy and not-serving answers do not excuse it). With fetch_fallback=sync: 3 failed fetches, or 10 excused ones, → Fallback, and the whole bootstrap ends in Fallback at fetch_wait + 30 min + the agreed snapshot's download time (at fetch_rate_kbps when set, else at 64 KB/s for at most 2 h) whatever the counters say; with wait it keeps retrying. The whole-file SHA-256 is checked from the next tick, at most 64 MiB per tick; on a mismatch the file is reset and the fetcher goes back to Collect. fetch_rate_kbps limits the bytes requested; there is a progress line every 10 % and every 30 s.
    • Import: result() returns the manifest and the agreed sources. onImportFailed(file) refuses that file's (name, sha256) and bans every agreed source that declared it (with no file at fault: the source that signed the manifest), keeps the files already verified (they are re-checked) and goes back to Collect; after 3 refused imports it goes to Fallback with sync, back to Collect with wait. Agreement is recomputed only when its input changed. The 30-s line names a conflict between equal groups, the manifests not counted (address unknown) and the replies dropped; the fallback lines name CREDITS_SNAPSHOT=github. dryRun stops at Done after agreement or after fetch_wait. stop() clears everything outstanding.
  • Node::getSnapshotManifestReply / getSnapshotPieceReply forward to snapshotFetcher_ when it exists, otherwise they drop and count as before. Nothing constructs the fetcher in this step (P5 will), so a running node behaves the same.

Tests

  • snapshot_agreement_tests (16): netPrefix/netBlock forms, threshold, min_sources=1, same /24 counted once, trusted keys counted separately, IPv6 /48, unknown addresses do not count, duplicate key, 2 forged sources at S+1000 lose to 3 honest at S, equal support goes to the higher block, a configured key's group wins (but not below min_sources), only the lower block reaches the threshold, digest majority with the minority reported, tie as a conflict, different heads, shortfall.
  • snapshot_fetcher_tests (36): real SnapshotProvider instances as sources, serving five snapshots built by buildSnapshot from one generated chain. A1 and A2 are identical; B and E have one or two trailing bytes in walletscache; D has another digest. They cover all 15 cases of the plan: (1) discovery, (2) bad signature, relayed manifest, oversized chunk and an unsolicited reply, (3) 3 of 4 agree and the minority is logged, (4) piece spread and the 2-per-source / 8-total limits, (5) a corrupt piece, (6) a stalled source, (7) busy, (8) a source that leaves the neighbours, (9) a group that runs dry, (10) the whole-file mismatch, (11) fetch_wait with sync and with wait, (12) import failures over three groups and then Fallback, (13) dry run, (14) stop in each phase, (15) the rate cap at 200 KB/s. Two more tests cover the source table: 100 discovered keys from one /24 push out neither a configured key nor the sources of other networks, a full table makes room for new networks, 64 silent keys over 64 networks do not lock a newcomer out, and unknown addresses are not capped at discovery, have no vote until they are known, and are capped by the address resolved with the manifest. Further tests: a late neighbour and a source silent for its first two requests are kept; busy with no delay is not asked again before 500 ms; sources busy forever end each fetch after 5 min and the bootstrap after 3 with sync, never with wait; three waves of sources serving bad pieces are banned without reaching Fallback and an honest group completes; sources that only time out cannot prevent Fallback with sync; ten bad-data aborts end with sync; a source keeping the fetch alive with one piece every 4 min hits the cap, while an honest join limited by fetch_rate_kbps past 30 min completes; refused imports with wait keep looking; a slow and a fast source race without blame; a symbolic link in the download folder ends in Fallback; a slow downlink marks no source; slow sources go to one piece at a time and their late pieces are taken; a refused import bans the declaring sources, or the signer when no file is at fault; the conflict line.

🤖 Generated with Claude Code

EnzinoBB and others added 10 commits October 25, 2026 14:56
snapshot_agreement: netPrefix (/24, /48) and agree() over the collected
manifests (design 6.3): groups by (seq, head) and digests, highest block
first, trusted keys counted by key, shortfall for the waiting line.

SnapshotFetcher: discovery of serving peers and configured keys, manifest
requests every 30 s, agreement at each tick, download of the agreed
snapshot's files piece by piece (2 per source, 8 in total, 30-s timeout,
busy, marks, bad pieces, group switch, whole-file check, rate cap),
fetch_wait with sync/wait, import failures, dry run, stop. Single-threaded,
hooks for the network and the clock.

Node forwards the two reply messages to snapshotFetcher_ when it exists;
nothing constructs it yet (P5).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rce table

Agreement: among the groups that reach min_sources, a group with a
configured key first, then the most independent sources, then the highest
block (a tie-breaker only), so that min_sources forged sources at a made-up
later block lose to a larger honest group; equal groups are a conflict.
Every group not chosen carries the reason, logged with the minority.

Source table: configured keys (sources=, sources_file) have their own room;
discovered peers are limited to 4 per /24 (/48) and 64 in all; when full,
the oldest entry of the most represented prefix makes room, never one with
pieces outstanding or in the agreed group during a fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e manifest

Source table: a full table always makes room, whatever the prefixes: the
least useful discovered source goes first (not a neighbour, silent for a
minute, no manifest after a minute, then the most represented prefix);
discovered sources silent for a minute are let go, and an evicted key is
not taken again for 10 min, so that returning keys cannot churn the table.
Discovered sources are also limited to 16 per /16 (/32 IPv6).

Addresses: an address not known yet is not capped at discovery; it is
resolved again when the manifest arrives, and the per-prefix limits and
the agreement count both use that address. A source whose address is
unknown does not count towards agreement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…o symlinks, mode 0600

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…bans, logs)

- Silence: counted from the first unanswered request (two requests over more
  than 35 s), not from the discovery; asked again after 10 s twice, then
  every 30 s; no proactive eviction: silence is only an eviction tier when
  room is needed; the cool-down after an eviction is 60 s.
- Busy: delay floored at 500 ms (cap 10 s); busy for a minute without a
  piece leaves the source alone for a minute, without counting to a ban.
- A fetch always ends: no new byte for 5 min returns to Collect, and every
  return to Collect counts as an attempt with the refused imports
  (3 -> Fallback).
- A refused import bans every agreed source that declared the refused
  (name, sha256); without a file at fault, the source that signed the
  manifest.
- A manifest reply over the prefix limits no longer erases a source in use.
- Agreement recomputed only when its input changed; filesHash once per
  source when choosing the files.
- Whole-file SHA-256 at most 64 MiB per tick.
- Logs: the 30-s line names a conflict, the manifests not counted (address
  unknown) and the replies dropped; the fallback lines name
  CREDITS_SNAPSHOT=github; fetch_wait=0 documented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e a sync from 0

Piece timeout: 30 s plus the bytes outstanding to the source at 64 KB/s. A
source that timed out but delivered a piece in the last 60 s is not
marked: its piece is asked elsewhere and it gets one piece at a time until
it delivers two in time; its late reply is still taken (hash-checked) while
the piece is missing. A source that never delivered keeps the mark.

Attempts: refused imports and failed fetches are counted apart. A fetch
that ends with sources banned or marked in it goes back to Collect without
counting; other failed fetches count, 3 -> Fallback with fetch_fallback=sync,
never with wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d bootstrap

A failed fetch is not counted only when a source was banned in it for bytes
that do not match the signed manifest; timeouts, busy and not-serving
answers no longer excuse it. Excused aborts have their own cap (10, then
Fallback with fetch_fallback=sync). With sync the whole bootstrap ends in
Fallback at fetch_wait + 30 min whatever the counters say; with wait it
keeps going.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…view fixes

- With fetch_fallback=sync the bootstrap cap grows by the agreed snapshot's
  download time: at fetch_rate_kbps when set, else at 64 KB/s for at most
  2 h; a warning at the start of the fetch when the expected time exceeds
  what is left.
- With fetch_fallback=wait refused imports go back to Collect instead of
  Fallback (the declaring sources are banned).
- Late pieces: stored with their send time; for one more timeout they
  count towards their source's outstanding pieces; the source is not given
  the same chunk again while another is free; the oldest entry is evicted.
- An in-flight piece filled meanwhile by another source's late reply is
  dropped at expiry without marking anyone.
- Tests: the configured rate beyond 30 min, refused imports with wait, the
  re-assignment race, a symbolic link in the download folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB
EnzinoBB marked this pull request as ready for review October 25, 2026 16:02
@EnzinoBB
EnzinoBB merged commit 0be88cb into master Oct 25, 2026
2 checks passed
EnzinoBB added a commit that referenced this pull request Oct 25, 2026
Step P5 rewritten against the code merged through #104 and PR #103 (official
even nodes in CI): the init split into initCommon/initChain/initRunning, the
gates (chooseMessageAction, onNeighbourAdded, onPingChecked, onRoundTimeElapsed),
the worker thread for the import and initChain with the hand-over through
CallsQueue, the fetcher hooks, the hash databases released before the import,
the interrupted-import marker, the start-up verification with exit code 4, the
run.sh variables and the robustness rows with official even nodes, and the
risks with a test for each.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant